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Research Brief 



Supplemental Educational Services (SES) Provision of 
No Child Left Behind 



Major Findings 

• SES programs have a 
small, but statistically 
significant, positive 
impact on student 
achievement. 

• Average effect size 
estimates were +.043 for 
math programs and +.017 
for reading programs. 
These estimates are based 
on a synthesis of more 
than 400 individual 
provider effects derived 
from rigorous studies 
involving over 140,000 
students. 

• Previous Title I 
intervention programs 
have had substantially 
higher average effect 
sizes than the SES 
program. 

• Four characteristics 
were found to be 
indicative of more 
effective programs. 

• SES programs provided 
by school districts had 
larger positive effect 
sizes than those offered 
by outside providers. 



Abstract 

The 2001 renewal of the United States’ Title I program 
instituted the Supplemental Educational Services (SES) 
program, in which schools in their third year of failing to 
make Adequate Yearly Progress (AYP) are required to offer 
out-of-school-time tutoring in core subjects to low-income 
students. This study synthesized provider effects reported in 
the extant body of SES provider evaluations to generate an 
estimate of the overall effectiveness of the SES policy in 
terms of improving student achievement and to identify 
provider characteristics that are associated with variation in 
student achievement effects. 

Background 

The 2001 renewal of the United States’ Title I program, 
which provides federal funds to schools with large 
populations of low-income students, instituted the 
Supplemental Educational Services (SES) program. The SES 
program requires schools in their third year of failing to 
make Adequate Yearly Progress (AYP) to offer out-of- 
school-time tutoring in core subjects to low-income students. 

This program, an integral part of the No Child Left Behind 
(NCLB) legislation reauthorizing the Elementary and 
Secondary Education Act, allows either public or private 
agencies to provide tutoring services. SES provides funding 
for out-of-school or extended-day tutoring services to 
students enrolled in targeted schools. Funds to pay for SES 
are allocated from Title I funds and account for up to 20% of 
these funds. Annually, more than half a million children 
participate in SES. Additionally, hundreds of tutoring 
agencies have been approved to provide services across the 
nation. 
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Much of the responsibility for SES programs lies with individual states. These responsibilities 
require states to approve, oversee, and evaluate their SES providers. Aceording to NCLB 
legislation, states must withdraw those providers from the approved list who failed to provide 
evidenee of improved academic achievement of students served for two eonseeutive years 
(USGAO, 2006). This process has proved difficult, however, as there are neither federal 
regulations nor funds for evaluating SES providers. 

Nevertheless, a number of state and loeal school districts have eondueted rigorous empirieal 
evaluations of provider effects since 2003. There are now enough individual provider effect 
estimates from high quality studies to permit a synthesis of effects to gauge the overall 
effectiveness of SES policy in improving student achievement, and to identify provider 
eharaeteristics that may be associated with variations in student outcomes. 

The purpose of this study was to analyze this information by synthesizing and modeling provider 
effects to estimate the effectiveness of SES as a whole, to inform the design of effective 
programs, and to assist in the development of scientifieally-based criteria upon which to base 
approval, removal, and continuance decisions. 



Study Methodology 

To conduct the analysis, Hedges and Olkin’s (1985) meta-anal 3 ^ic parametric estimation of 
effect sizes methodology was used to estimate mean effect size estimates for math and reading 
outeomes and to identify eharaeteristies of SES providers assoeiated with variation in effects. 
Studies incorporated in the meta-analysis had to meet the following inclusion criteria: 

• Reported a sample size (n); 

• Reported an effeet size (Cohen’s d) by provider or eontained sufficient information 
to compute an effeet size; 

• Restrieted the analyses to students who reeeived at least 15 hours of SES tutoring. 

• Compared SES student performance to that of a comparison group of SES-eligible 
students attending the same sehools; 

• Measured student achievement with a valid and reliable standardized test; and 

• SES provider eharaeteristics could be gleaned from the report or other reliable sources. 

Results 

Overall, 400 math effects and 401 reading effects were collected and analyzed, yielding a grand 
total of 801 effects. The math and reading effeets were analyzed independently to determine an 
overall mean effect size for SES on the outcomes in each subject area. Eor the math analyses, 28 
SES evaluations were included from 12 states or loeal sehool districts. For the reading analyses, 
27 SES evaluations from 1 1 states or local school districts were included in the meta-analysis. 



The mean weighted math effect size was -I-.043. The number of students included in these studies 
was 140,846. Despite being quite small, the 95% eonfidenee interval indieated that the ^.043 
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effect size was statistically significant. The mean weighted reading effect size was +.017. The 
number of students included in these studies was 139, 844. The test for statistical significance at 
the 95% confidence interval indicated that the overall mean reading effect, though also very 
small, was statistically significant. 

Generally, effect sizes of .20 are considered small, effect sizes of .50 are considered moderate, 
and effect sizes of .80 are considered large (Cohen, Manion, & Morrison, 2000). However, in the 
context of previous educational research, much smaller effect size estimates might be expected 
(Hill, Bloom, Black, & Lipsey, 2008). Even in this light, the effect size estimates revealed in this 
analysis are very small, particularly the reading outcome. 

To appropriately interpret the effect sizes reported in this study, they should be placed in the 
context of the effectiveness of other Title I reform efforts and syntheses of tutoring effects. Meta- 
analyses of tutoring programs over the last quarter century have revealed much larger effects of 
tutoring on student achievement. For example, a meta-analysis of tutoring programs conducted 
by Cohen et al. (1982) revealed effect sizes of .29 for reading and .60 for math. A meta-analysis 
of volunteer tutoring programs conducted by Ritter et al. (2009) found a similar overall effect 
size for reading, .26, and a .27 overall effect size for math tutoring. An examination of out-of- 
school-time instructional efforts conducted by Lauer et al. (2006) found a .07 overall effect size 
for reading and a .16 overall effect size for math programs both of which are much lower than 
Cohen et al.’s (1982) and Ritter et al.’s (2009) findings but still substantially higher than what 
was found here. A comparison of tutoring effects from meta-analyses is illustrated in Figure 1 . 

Figure 1. A comparison of tutoring effects by subject area from meta-analyses. 



Comparison of Tutoring Effects from 
Meta-Analyses 




Chappell et al. Cohen et al. Ritter et al. Lauer et al. 
(SES,2009) (1982) (2009) (2006) 



■ Reading Effect Size 

■ Math Effect Size 



Study 



SES also appears to be far less effective in improving student achievement than previous 
approaches supported by previous Title 1 policies. For example, a meta-analysis of 
comprehensive school reform (CSR) efforts revealed an overall effect size of +.15 (Borman, 
Hewes, Overman, & Brown, 2003), which according to Hill et al. (2008) could be interpreted as 
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a moderate to large program effect. The authors of that study concluded that CSR was positively 
impacting achievement on a school-wide basis and that effects were larger the longer the 
programs were in place (Borman et ah, 2003). Similarly, a meta-analysis of all Title I programs 
from Title Fs inception (1965) until 1994 revealed an overall effect of -i-.l 1, which also can be 
interpreted as a moderate effect (Hill et ah, 2008) and is much higher than the effect sizes of the 
SES program found here. Figure 2 compares the effect sizes of SES to other Title 1 reforms. 



Figure 2. A comparison of effect sizes of Title I school effectiveness meta-analyses. 
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Analysis of Provider Characteristics 

Further analysis revealed that the effect sizes found here were not consistent across studies for 
either the math or reading subject areas. This lack of consistency indicates that variance in the 
effects across studies may be attributable to other variables (Hedges & Olkin, 1985). In other 
words, certain SES provider characteristics may help explain the differences between outcomes 
for these studies. 

For the math analyses, the results revealed that providers exhibiting the following characteristics 
had larger mean effect sizes: 

• Used a prescribed curriculum; 

• Employed only tutors with 4-year degrees; 

• Offered English Language Learner (ELL) services; 

• Provided Special Education (SPED) services; 

• Supplied initial training to tutors; 

• Offered on-going training to tutors; 

• Provided both math and reading subject tutoring; and 

• Supplied 1:1 tutoring (as opposed to small-group tutoring). 

Figure 3 illustrates the effect sizes for providers who exhibited these characteristics versus 
overall effect sizes and effect sizes for providers who do not share these characteristics. 
Consistent with findings from previous research on math tutoring, providers offering small group 
tutoring tended to have larger effect sizes than those offering 1:1 tutoring. (Juel, 1996; Lauer et 
al., 2006). 
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Figure 3. A comparison of math effect sizes by provider characteristics. 



Math Effect Sizes by 
Provider Characteristic 
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For the reading analyses, the results revealed that providers exhibiting the following 
characteristics had larger mean effect sizes: 

• Employed only tutors with 4-year degrees 

• Offered Special Education services 

• Provided both math and reading subject tutoring 

• Utilized 1 : 1 tutoring (as opposed to small-group tutoring) 

Figure 4 shows the effect sizes for providers who exhibit these characteristics compared to the 
overall effect sizes and to effect sizes for providers who do not share these characteristics. Other 
provider characteristics revealed either negligible or negative effect sizes. 
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Figure 4. A comparison of reading effect sizes for tutor characteristics. 
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School District Providers 

For both the math and reading analyses, local school district providers, when compared to other 
SES providers, had a higher mean effect size (math effect size = .094, reading effect size = .024) 
than the overall mean effect size of each subject area from their commercial counterparts. Figure 
5 illustrates the differences among the three effect sizes for each subject area. These results 
indicate that, on average, local school district providers have a more substantial impact on 
student achievement than commercial or other non-profit providers. 

Figure 5. A comparison of ejfect sizes for local school districts V5. commercial Providers. 
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School district providers may show greater effect sizes due to characteristics inherent in district 
composition and service delivery. These eharaeteristics include using tutors who hold at least a 
four-year degree, using a prescribed curriculum, offering tutoring in both subject areas, and 
offering services to ELL and SPED students. All of the distriet providers ineluded in the analysis 
possessed each of these characteristics. A review of the literature on tutoring also indicates 
several of these same traits as essential for a sueeessful tutoring program (Cohen et al., 1982; 
Fashola, 1998; Topping, 2000; Wasik, 1998). 

Conclusions & Recommendations 

The legislative intent of the SES program is to narrow or elose the aehievement gap by 
improving the academic achievement of historically underperforming populations. Our findings 
indieate that the overall effeet of SES on student aehievement is quite small when compared to 
previous Title 1 reforms and previous studies of tutoring effects. 

The results of this study provide some guidance for design and approval of SES tutoring 
serviees, which, consequently, are consistent with deeades of prior research on tutoring. 
Characteristics of effective SES tutoring programs identified in this study include (a) the use of 
school district providers; (b) experieneed, well-trained tutors with four- year degrees; (c) a 
national or prescribed curriculum; and, (d) one-to-one tutoring for reading instruction. Effect 
size estimates for both the math and reading analyses were higher for providers that had these 
traits. 

Although the evidence presented here provides some guidance for structuring and approving SES 
provider programs, the small overall effeets associated with SES suggest that, as a policy, SES is 
not having the desired effect. In cases where school districts were granted an exception and were 
allowed to offer their own SES programs, the sehool districts were three times more effective in 
increasing math achievement relative to other providers. School district programs also were 
offered at a fraetion of the cost— the costs of providing SES are all marginal for sehool districts, 
whereas private and non-profit providers incur additional fixed costs. 

As Congress considers re-authorization of the Elementary and Secondary Education Act, we 
suggest a eareful review of the Supplemental Edueational Serviees provision in light of these 
findings. Despite mounting evidence that SES is far less effective than previous Title I policies, 
we are not aware of a single instanee in which a provider has been removed from an approved 
state list on the basis of failing to demonstrate positive effects on student achievement. 
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